Is AI changing the way we think?
In this edition of The Naked Scientists, from bogus scientific papers and misleading made-up “facts”, to potentially curtailing our critical thinking, we look at the effect that mass adoption of AI might be having on the way we think, the decisions we make and the information we learn and act on…
In this episode
How did AI explode in popularity?
Mike Wooldridge, University of Oxford
Artificial intelligence is being talked about and used more widely than ever. Rapid advancements in this technology have, for example, led to many remarkable and welcome breakthroughs in science, and major efficiency gains in some workplaces. It seems certain that the widespread adoption of AI is an inevitability, much like the internet and social media. But, over recent months, AI concerns have come to the forefront, both for businesses and individuals, and that’s what we’re going to explore this week. One of the main ways that the majority of us are running into AI is via platforms like ChatGPT; these are so-called “large language models”, which specialise in understanding instructions we provide in plain English and then rapidly generating summaries, explanations or even diagrams and computer code to order. And it is very impressive. But all that glitters is not gold. Because these systems make a lot of mistakes; and as a lot of Internet content is now being generated this way, we’re running the risk of polluting the knowledge space with rubbish that could potentially be around as a misleading distraction for decades. Here’s Mike Wooldridge, Professor of the Foundations of Artificial Intelligence at Oxford University…
Mike - Large language models are quite a recent development. They've really only been around in the way that we recognise them now for about half a dozen years. And what they do fundamentally is something which everybody who's listened to this programme will be familiar with. If you open up your smartphone and you start sending a text message to your mother or your partner and you say, “I'm going to be…” it will suggest completions. And how your phone is doing that is it's been trained on all of the text messages that you've sent. And it's learnt that the likeliest next thing to come after “I'm going to be,” is either “late” or “walking the dog,” or in my case, “in the pub.” Large language models have been trained, not just on a bunch of text messages, but essentially all of the digital information that's available in the world, essentially all of the worldwide web, which is an unimaginably vast quantity of text. And it is really just ordinary written human text. So every web page is scraped and all the text that's on that page is extracted from it. And all of that is fed to the enormous neural networks that are in large language models, which are trained so that they can make the best prediction possible about what should come after what we call your prompt. And your prompt might be, ‘a one-paragraph summary of the life and achievements of Winston Churchill,’ or ‘the recent performance of Liverpool Football Club in the European Champions League,’ or whatever. But all of that text, the whole of the worldwide web is absorbed by these vast, vast neural networks in a process that takes typically months, and requires millions of pounds worth of computer processing power. And what we end up with is something that's really quite remarkable.
Chris - And when these things are roving across the internet, hoovering up that information, is that indiscriminate? Or would it do it like you would do it? Which is you'd say, oh, here's a web page from the University of Oxford. I set that as having greater store and veracity and validity compared to a web page that was produced on someone's blog five years ago.
Mike - There is a certain amount of discrimination that goes on, but actually the demand for the data is so great. You really need absolutely all the data that you can throw at it, that it ends up being somewhat indiscriminate just because the demand for that data is so vast. The scale of the data is really, really astronomical. The breakthrough large language model was called GPT-3, and it was released in 2020. There were 40 terabytes of ordinary written text used to train that model. That's the equivalent of millions upon millions of ordinary books. And if your demand for that text is so great, then there's a limit to how discriminating you can be. But interestingly, the evidence is the higher quality text that you're able to train your model on, the better it's going to be. So there is a market right now for that text. There are also, I have to say, some pretty unscrupulous practices that are being used by some of the large language model builders who are, if not actually breaking the law, then probably running roughshod over copyright restrictions. And there are lawsuits underway right now about that.
Chris - So if these things are just ingesting things willy-nilly and looking for connections, if you've got these engines producing this material, which then ends up on the internet, are they going to end up force-feeding themselves their own stuff? Is that like me eating my own excrement?
Mike - Yes. Thank you for that, Chris. Let me put it this way. It seems inevitable to me that within 20 years at the most, but probably more likely within 10, essentially everything that we read on the internet, on the worldwide web is going to be AI-generated, unless you're prepared to pay a premium for human-generated content. And yes, you're absolutely right, what that means is that this is the content that models are then going to be trained on. And the evidence at the moment is that models which are trained on AI-generated content are poorer. And if you just iterate that process over a few generations, it can quite quickly descend into nonsense. So this is one of the big conundrums of the age in AI. Where are we going to get that future data from? And how are we going to ensure that models aren't just trained on ‘AI slop,’ as it's now called. But also what this signals is there is going to be a market for human-generated content. Data which is written text or whatever kind of content that's created by human beings will remain extremely valuable.
Chris - The other thing that we've heard about is that these models do tend to make stuff up. So if they are just stringing together material that they have gathered from across the internet, why do they make stuff up?
Mike - These models have no conception of what's true or false. They're simply designed to make the best guess possible about what should come after your prompt. And in particular, in the absence of any information, they will fill in gaps. But the weirdly troubling thing about this is they will fill in those gaps in extremely plausible ways. And that means that not only do they get things wrong, they get things wrong in ways that are actually quite difficult for us to detect. So here's my favourite example. An early large language model that I tried out, I asked it about me. And when we finally got to the Michael Wooldridge that wasn't the Australian Health Minister or the BBC News reporter, it said, "Michael Wooldridge, Professor of Artificial Intelligence," tick, "known for work on multi-agent systems," tick. And then it said it "studied at the University of Cambridge." And I never studied at the University of Cambridge. I never had any affiliation with the institute. So why is it getting it wrong? Because actually for an Oxford professor, that's quite a common background. And in the absence of any information, it's kind of filling in the gap in a very plausible way. But if somebody read that, they wouldn't notice that because it sounds plausible. So they get things wrong a lot and they get things wrong in extremely plausible ways. And that makes it very hard to detect those falsehoods. You may have heard recently that GPT-5 was released. So GPT-3 from OpenAI was really, for me anyway, the breakthrough large language model. That was really eye-opening. It was a huge step change in capability over its predecessor. And then GPT-4 released about two years after that was also a step change in capability. The one thing I think that would have made the biggest difference in GPT-5 was if it was better at not producing falsehoods. The problem is it still seems like that's a really, really big issue. And if I had to identify right now the single biggest barrier that's in the way of the wider take-up of these models, it's this fact that they just get things wrong so often. They get things wrong a lot and they get things wrong in very plausible ways.
09:26 - AI generated research threatens to pollute the corpus
AI generated research threatens to pollute the corpus
Jennifer Wright, Cambridge University Press & Marie Souliere, Frontiers
AI can make research faster and more in-depth when used correctly, but AI confabulation like claiming Patrick Moore created the Naked Scientists, could have serious impacts on the quality of scientific literature. We are already seeing studies and review articles written using AI ‘essay mills.’ These are AI generated pieces of writing that also contain completely made-up information. They’ll contain invented references created to support a particular line of argumentation, or “facts” that are just plain wrong. But they look and read so well that the scientific wheat can be hard to separate from the chaff. Some of these studies are definitely slipping through the cracks and getting into journals. In other situations, people are generating articles with AI and then sending them to paid-publication outlets, where others may rely on them without realising that a human didn’t create or check them. Other AIs may also then ingest and regurgitate this same information. This challenges the integrity of published studies and the trust between everyday people and scientific information. To find out how trust in the scientific community can be maintained in the age of AI, here’s Jennifer Wright from the Cambridge University Press, head of publication ethics and research integrity…
Jenny - When you're a scientist you look at someone else's research and you think, is this good? Is this really correct? Is there something that they've missed? Is there something they should have accounted for? Is that method the right approach? And this is essentially what we do today through a process called peer review. So experts in a topic take a paper, they look at it, they grill the work, they critique it and then they determine whether or not it's suitable for publication at all or maybe whether there are some improvements that the author or researcher could make to make it suitable.
Chris - And obviously your venue has a reputation to defend but not all venues do that, do they? Not all publishers operate the way Cambridge does. There are some where I just send them a very large cheque and they publish anything. I've come across these on the internet.
Jenny - Unfortunately this is becoming increasingly challenging. There are so-called 'hijacked' journals which are essentially really sophisticated spoofs of real journals. So let's say you had the ‘Journal of Interesting Research’ and that's a legitimate journal by a legitimate publisher. There might also be another ‘Journal of Interesting Research’ and that is a predatory journal or a hijacked journal and they can be very sophisticated, so researchers and readers might not even realise that they're looking at a predatory journal.
Chris - What's the purpose of those places existing though? Is this people burnishing their CV? So if I want to get some extra publications and make myself look like a better scientist than I am, I can generate some content and send it to one of these journals who ask few questions but will just generate the publication and that turns into CV points for me. Is that the purpose?
Jenny - There are probably a lot of motivations, as with anything like this. It could be naivety, it could be that the researcher doesn't realise this isn't a legitimate outlet, it could be pressure to publish, which you've alluded to, that a lot of academic careers are built on your publication record. So a shortcut to a publication record might be attractive to certain people in certain parts of the world as well, where they have really strict requirements around, for example, you can't graduate until you have a paper, you maybe can't get a promotion until you have a paper. So with that incentive structure it becomes very difficult to resist perhaps.
Jennifer Wright. Bogus research and predatory journals are nothing new, but recent advancements in generative AI have turbocharged the ease with which content can be produced, with plausible papers now popping up at the click of a button. These studies are often littered with confabulations, which unsuspecting readers might mistake for legitimate findings. Here’s Marie Souliere, Head of Editorial Ethics and Quality Assurance with the open access publisher Frontiers…
Marie - The real concerns are about inaccurate or false content, what we refer to as hallucinated content from the artificial intelligence, hallucinated references. If the AI does plagiarism, if there is poor attribution of content without referring to the real source, this is actually what we are really concerned about rather than AI being used for the benefits that it can provide because it is a very efficient and effective tool for analysis of research data. It can free up time for researchers to carry out more research and it really, really supports non-native English speakers, who are the majority of the population of researchers in the world.
Chris - Given that it can churn out very slick, very nice-looking content that's extremely plausible and could therefore deceive reviewers - because unless they're going to check everything in excruciating detail, things could slip through - does that not worry you? That this is going to create a lot of work for reviewers to be able to say, "Honestly, I've really, really checked this and I've checked every reference, I know absolutely this is rock solid.” Which, let's face it, no one's got enough time to do exactly that.
Marie - Agreed and what we've been doing is in a way fighting AI with AI. So there are a lot of tools that have been created to support publishers in the industry and everything is focusing on the root cause of the problem. We cannot detect AI-generated content, it's very difficult and there's very low accuracy for this. So what we do is focus on everything else that might be wrong with the article that could be a symptom of a fake paper. So we're looking at tortured phrases, some problematic content, data that looks a bit dodgy, gibberish images, wrong attributions so that they don't get published and become part of the literature.
Chris - That is the case but there are lots and lots of journals, they don't have the kinds of checks and balances that you have and the problem that stems from that is the AI platforms that people are using and training are just romping their way around the internet, sucking all this stuff up and incorporating it into the model that they then use to generate papers that do go into legitimate sources and reputable venues like those that you publish and so it does have the potential to build up like a layer of plastic on the seafloor and pollute the knowledge space for years to come, doesn't it?
Marie - Absolutely and this is a frequent concern that's raised by researchers and publishers alike. I was at the Frankfurt Book Fair last year at a big event, there was a whole AI day with publishers, people were talking about how these non-peer-reviewed articles in archive or in predatory publishers are taken with the same level of legitimacy as peer-reviewed articles and AIs are not taking this into consideration, or the developers are not, because these articles, the real ones versus the archives, are tagged with a little label that says it's been peer-reviewed and the AIs should be able to make a distinction, but not with the predatory publishers and that's the biggest risk.
Marie Souliere from the Frontiers publishing house. Increased vigilance from publishers, editors and researchers is crucially important to combat this problem, but another way to beat AI pollution of the knowledge pool could mean changes to the way scientists document their work. Here’s Jennifer Wright from Cambridge University Press again…
Jenny - We also promote open science principles at Cambridge, so you can think of this like the show-your-working mode of research. It's not enough to just present the outcome, the researcher needs to be able to kind of show the journey, ideally whilst they're doing it, but a paper trail of: this was my lab work, this is the microscope I used, this is when I went to the field - that's much harder to fabricate than some words on a page. So I think that's something else that builds trust, being able to see that journey. It's also about collaboration across publishers, institutions, funders, libraries, all coming together to look at what can we really do about this pressure-to-publish culture, incentive structures, how can we turn down the tap rather than just filtering the water. So Cambridge is working on this just now. We've got a white paper coming out in the autumn which is going to look at some of these big systemic challenges and what we can actionably do about it.
18:38 - 'Sycophantic' AI might be responsible for mental health harm
'Sycophantic' AI might be responsible for mental health harm
David McLaughlan, Priory Hospital Roehampton
In recent months, doctors and nurses have seen a rise in patients being admitted for AI-related issues. In one extreme example, a 60-year-old went to hospital in the United States claiming that he was being poisoned by a neighbour. He had been using sodium bromide instead of salt - sodium chloride - on his food for 3 months after asking ChatGPT to suggest “salt alternatives” to lower his blood pressure. He developed “bromism”, which can cause hallucinations, and took several weeks to recover. People are also using ChatGPT for mental health advice and companionship. As a result, a new term “AI psychosis” is being used to describe users of AI chatbots who are losing touch with reality as a result. Some speculate that the immersive nature of the interactions we can have with these systems make them provocative stimuli that can trigger the emergence of a psychotic state in some vulnerable people. I’ve been speaking with consultant psychiatrist at the Priory Hospital Roehampton, David McLaughlan...
David - Recently, one of the things that journalists have been asking about is AI psychosis, because there have been a couple of case studies recently where members of the public have developed psychosis involving conversational AI like ChatGPT. So people have believed that they're speaking to a real person, or they've developed paranoid delusions that have involved chat GPT or other generative AI. And that's really excited journalists recently. There have been lots of headlines about AI psychosis.
Chris - Do you think this is a new phenomenon? As in, this is a new risk, a new outcome, and it's because of a new technology? Or do you think that these people are always going to be vulnerable and it's just this that's doing it rather than 20 years ago, it would have been the television or the radio that was provoking this?
David - Exactly that. One of my pet peeves has been the media misrepresentation that this is a new condition, that this is a new disease or new illness that we all suddenly need to pay attention to. Psychosis is an illness where people develop hallucinations. So a hallucination is when people perceive a sensory stimulus that isn't there. So they might hear a voice or they might even have visual hallucinations, seeing things that aren't there. This illness, psychosis, has always existed. The underlying neurobiological abnormalities, so that's the the differences in the brains of people that have psychosis, hasn't changed. It manifests in different ways according to the world that we live in.
The things that we become delusional about take on the themes of the environment which we're based in. So perhaps 70, 80 years ago, when televisions first were invented, people would have developed delusional beliefs that television was talking to them. There's something called Truman Show Syndrome. After that film came out people were often presenting to clinics and hospitals, like the hospitals I've worked in, believing that they were being followed by cameras. But it's not the television that caused the psychosis. It's not the radio that's caused the psychosis. It's not the film, the Truman Show, that's caused the psychosis. And in this case, again, it's not generative AI that has caused the psychosis. It's just a theme in which this condition presents itself.
Chris - Is it not potentially a more provocative stimulus to a person potentially developing a psychotic state? Because it will basically, through that person training it to do so, learn to push their buttons more effectively with time. It might be the outcome's going to be the same, whether it's the TV or the radio causing psychosis. Perhaps though, what we're going to see is a faster route to a psychotic state through these systems, because they're so good at finding out what floats our boat psychologically.
David - It's in the interest of the developers to create that kind of dynamic where the conversational AI says what it thinks you want it to say. And in clinical terms, there is a danger or risk that you get something called collusion. And collusion is when somebody around you reinforces delusional beliefs that you have. What I would see with families or friends is often it's actually to avoid conflict. I might have a mother who's concerned about her son and some paranoid delusional belief that he has about people following him home from school. Rather than creating an argument, she just agrees with him because it's easier for her to do that. And in a clinical setting, what I would normally ask a family or friend to do is to gently challenge those delusional beliefs. Not to create an argument, not to create conflict and shut down the relationship, but to gently challenge that. I would hope that that's what generative AI does and that it's not entirely sycophantic. But I'm not sure if that's always the case.
Chris - But at the same time, if you think back about 10 years ago or so, Julian Leff working down in London was pioneering the use of avatars and creating technological representations of a person's voices, for example, that they were hearing as a way for them to challenge and push back. So it's almost like we need to tweak how these engines work because we could actually turn them from something that could make a disease state into something that could help to remedy a disease state if they were programmed the right way.
David - Exactly. I'm really familiar with that research. So patients with illnesses like schizophrenia who are hearing these voices, it's almost like training for them, teaching them how to ignore or dismiss or challenge these auditory hallucinations that were sometimes telling them really horrible things about themselves or asking them to do horrible things. The technology itself has enormous potential and we shouldn't always be afraid of technology, but it's more how it's used and just to always take a critical mind. That's what I learned when I was a research fellow, was always to be critical of information that was presented to you as a fact and always to challenge inherited wisdom and to take that critical mindset. And I think if we continue to do that, then I think we're safe to keep working with technology.
Could using AI cripple our cognition?
Sam Gilbert, UCL
We are, of course, using AI technology every day. Large language models like ChatGPT can summarise texts, draft emails and find any information on a topic it can scavenge from the internet, all within mere seconds. Things that people used to do themselves are being outsourced to AI to save some time. But research suggests that by avoiding these tasks that keep the brain ticking, AI use is having a real impact on our cognitive function. Is AI making us stupid? I spoke with Sam Gilbert, Professor of cognitive neuroscience at UCL, to find the answer.
Sam - Learning is a physical process that takes place in our brains. When you learn something your brain changes, but it's not just about what happens inside your brain. I think learning is a process that extends into our environment as well. We think not just with our brain by itself, but as part of a coupled system of a kind of a duet between what's happening inside our brain and what's happening outside it. So in my opinion, learning and memory is a process that extends into our environment. It's not just about what happens physically inside our brains.
Chris - So the things in the environment also become part of that process. If you're a musician, for example, your instrument is also part of the memory process of learning to play a piece of music.
Sam - Exactly. You're a coupled system.
Chris - When we train people to learn, though, if you think about how we used to teach kids times tables and they'd learn by rote, and then people decided that was unfashionable, and we taught other people different ways of thinking about maths and so on. Does then potentially introducing a tool like AI or ChatGPT, to name one of them, have an impact on the way people think in the same way that stopping learning times tables or doing spelling tests at school has affected the way that people think about maths and language?
Sam - I think it certainly does change the way that we think and the way that we remember.
One of the shifts that people talk about is a shift from storing information or storing knowledge inside our brain to instead storing where to find that information. So it's a shift into thinking about critical evaluation and how to locate information rather than storing the information itself. So that can be a good thing. It can make people more critical. It can make people more engaged in the material that they think about. It can lead to problems. For instance, if you store some important information in a device and that device breaks or the battery dies, then you might be in trouble. So there are some complex pros and cons of using these devices and it's not as simple as good or bad.
Chris - But if it can alter the fabric of our brain, and I'm thinking back to the study on London taxi drivers, if they learn the knowledge of London, all those thousands of streets, they get a bigger part of the brain, the hippocampus, that we know processes that kind of information.
So if a person's offloading a lot of their cognition onto ChatGPT, are they bending their brain? Are they potentially getting less well-practised at writing summaries and abstracts for their science papers or writing short stories and so on?
Sam - So there are two complementary phenomena in this field. One of them is called the Google effect. I don't know why it's called the Google effect. It's not really got anything to do with Google, but that's the name that stuck. So the Google effect refers to the way that when you store your memories in an external device, you then tend to be lost without that device. You can't remember it anymore if the device fails. So that sounds pretty bad. But on the other side is something called saving enhanced memory. And that shows that when you offload some of your memories into an external device, then it actually helps you remember additional unsaved information that you might not have remembered otherwise.
Chris - Which do you think, though, is the best for long-term brain health? Because the other thing we're very concerned about with an ageing population, more people finding themselves living into the era where dementia becomes a lot more common. And there are some schools of thought that argue we have a sort of ‘use it or lose it’ thing. If you rehearse your brain regularly, do the crossword every day, et cetera, this can help to maintain cognition for longer. So do you think there's a risk if we do offload too much onto these devices that we're not getting that ‘use it or lose it’ stimulus that's going to help us age better?
Sam - I think the evidence points in the other direction, in fact. So there have been a few studies where people have measured how much people use the internet and other forms of digital technology and then gone on to measure whether people go on to develop dementia afterwards. And in fact, the evidence is quite clear that the people who use more digital technology at an earlier point in time are actually less likely to develop dementia afterwards. Now, it's hard to interpret these studies. It could be there are confounding factors that might be involved here where people who have more access to these devices may have more money or there might be other reasons involved in this. But if it was the case that we were causing catastrophic damage to our mental abilities by using technology too much, you would predict exactly the opposite finding. And that's definitely not what we find.
Chris - And what about at the very young end of the spectrum with learning and building a brain that will be fit for adulthood? Is there a risk that children don't build enough brain muscle at a young age and, as a result, it will retard their ability to do things in future?
Sam - It's definitely very important that we think about the educational activities and how those interact with technologies that are available. And the important thing here is to incentivise the kinds of mental activities that we want to promote. So that's not necessarily rote learning or doing the things which technology is very good at doing already. And instead, we need to be focusing on the things which technology still is not optimal at helping with. That's things like creative thinking, evaluation, synthesis of large amounts of information in a balanced way. These sorts of activities will remain important in education and throughout life. I work in a university, so the undergraduate students that I teach are increasingly using tools like ChatGPT and so on. And I think this clearly poses a challenge. I'm not sure that it's necessarily harming their learning, but it certainly changes the incentives that we need to offer to make sure that the students are using their brains in a way that is going to be most helpful in their later life.
Chris - I don't know if you're a betting man, Sam, but where would you put your money on this? Do you think that we're going to find out in 20 years time, do you think you're going to look back and say, yeah, I was right, there's not going to be an impact? Or do you think maybe there is, and maybe we should be looking?
Sam - Technology is not new. We worry now about things like ChatGPT, but of course, cognitive technology goes back at least to the invention of writing and earlier. And so just as these technologies are not new, the fears about technologies are also not new. Famously, Socrates warned that reading and writing would make people forgetful. And in my opinion, I see nothing in the current evidence that shows that contemporary technologies are any more harmful than earlier ones. Having said that, it's definitely a complex picture of benefits and potential harms. We need to keep our eyes open and think carefully about the ways that people use technology, rather than just saying technology as a whole is good or bad.
Comments
Add a comment